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The ye Do — They Do Model provides an evaluation 
methodology that seivice providers can use on a small scale to obtain 
data for themselves and for the over-all system. The model allows for 
coriplete assessment of ongoing program implementation and outcome. 
Three questions are used to organize evaluation activities: (1) Did 
staff (We) do what we said we were going to do? (2) Did participants 
(They) do what they were supposed to do? and (3) If everyone did what 
they were supposed to do, how would we know if it did any good? These 
questions allow for evaluation of program activities and outcomes for 
both formative and summative purposes. Jt focuses on staff 
activities, client activities, and the outcome o£ these activities. 
The Model allows for relating of outcomes to program goal activities. 
It is useful for establishment of objectives, identification of 
specific treatment activities, and establishment of assessment 
methodology. Key aspects of the Model are a focus on evaluation of 
treatment integrity, and a simple reporting method that service staff 
can use with minimal disruption of activities. Use of the model is 
illustrated in the evaluation of the University of Nebraska's 
Educational Center for Disabled students. Computer technology was 
used as an intervention to train disabled college students. 
Objectives included improvement in attitudes and in academic 
performance. (MAC) 
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ABSTIMCT 

This paptr providts an ovtrvitw o^ tht Ut Do - Thty Do program 
tvaluation modt 1 • Tht Ut Do - Thty Do Hodtl focusts tvaluation tfforts 
on sta^f activitifs, clitnt actsvitits, and tht outconts o^ thtst 
activitifft. Tht Hodtl it um^uI ^or tstabl ishmtnt oi object ivts, 
idtnt if ication oi sptcific trtatrntnt activitits, and tstabl ishmtnt oi 
asstssntnt mtthodologx. Sptcif ic txamplts oi tht ust oi t^f Hodtl art 
providtd. 



Uf Do - Thty Do 



UE DO - THEY DO: A MODEL FOR PiMCTICAL SERVICE PR0GRM1 BMLIMTION 
PimdoxicillXf fvaluation is o^ttn tisitr to accomplish on 4 
systM Midf as opposfd to individual program basis. In many casts 
fvaluation oi a school district or stattwidt strvict systtm can bt mort 
ffftctivfix impltrntnttd than tvaluation oi particular classrooms or 
strvicf dtlivtrx units. Thtrt art a numbtr oi rtasons why this paradox 
exists. Funding at tht systtm Itvtl can bt allocated ior evaluation 
activities, while individual program budgets rarely contain adequatti 
\i an?/| budget ior evaluation (DeStefano & Stake, 1986). Expert 
staffing, either in-house or by outside consultation, can be obtained 
ior evaluation system-wide, while individual programs must often rely 
only on existing service delivery staff who have little, if any, 
training or experience in evaluation activities. Large data-bases may 
exist at the system level that can provide information for sunmative 
statistical evaluation, while individual programs often must rely on 
qualitative data such as case notes. 

Differences such as these make evaluation of small scale 
individual programs difficult, however, the evaluation of actual 
service delivery at the program level may, in many cases, be more 
critical than overall system-n^ide evaluation. It is the direct 
interactions of service providers with clients or students that 
primarily determine client or student outcome. Activities at the 
service provider level often do not follow system-wide guidelines, 
leading to questions of treatment integrity that remain unanswered in 
system level evaluations. It does little good to evaluate a new 
curriculum at the district level if individual teachers are not 
following the curriculum. Similarly, it does little good to evaluate a 
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fttatt-widt Mrvicf program for a target population (t .g. mtntallx 
rttardtd or dt 1 inqutnt/prt *df 1 inqutnt) if tht individual provider units 
art all using difftrtnt trtatmtnt Mthods and proctdurts. 

In addition to trtatiMnt inttgritx issuts, thtrt art ofttn 
significant difftrtncts bttuwtn tht evaluation needs of the sxstem and 
the individual service units. Sxstem-wide evaluation is concerned wi th 
broad issues of cost-effectiveness, effectiveness of service deliver/ 
to population groups, general outcomes for target populations as a 
whole, staffing, budgeting and information flow within the bureaucracy. 
Stakeholders at the s/stem level include broad groups of funders, 
political units, and ccnmunitx groups for whom information must be 
generated in formal reports. Individual service units, on the other 
hand, need information on the effectiveness of dax-to*dax activities 
and on outcomes for individual clients. There is less need for 
budgeting and cost^eff ectiveness information, since the individual unit 
max have little or no control over these areas. Stake holders are much 
more limited in most cases, perhaps being limited to higher levels in 
the sxstem and the individual clients, for whom reporting max be onlx 
semi-formal or informal. 

While sxstem-wide evaluation can attempt to accomnodate these 
diverse needs, the sxstem-wide evaluator usual 1x can onlx sample 
individual provider units. Thus, there is little data available on 
dax~to-dax activities and on differences across provider units. To 
address the evaluation needs of service providers evaluation methods 
must be developed that can allow service provider units to conduct 
evaluation of their ongoing activities and generate data that is 
relevant to both themselves and to the sxstem. These methods must be 
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poMtrful tnough to allow tht collection o^ rtUvant data, x«t be simplt 
tnough to bt impl tut nttd by ptrtons with littit formal training and 
Mith littit tiiM for tvaluation activitits. 

Thf UE DO * THEY DO niodtl was dtvfloptd to providt an tvaluation 
mthodologx that can bt u^iliztd bx strvict providers to asstss ongoing 
program impltnifntation and outcomtft. Tht rtmaindtr oi this paptr will 
providt a description oi tht dfvflopmtnt oi tht Modt 1 , an overview of 
Modfl operation and txamplts drawn from the Educational Center for 
Disabled Students concerning use of the Model. 

The Educational Center for Disabled Students was established at 
the Universitx of Nebraska-Lincoln as a demonstration project through 
funding bx the U.S. Department of Education (CDFA 84.078C, Grant! 
G008530057). The Center provides computer technologx ^nd skill 
training to a diverse population of disabled college students 
(phxsicallx disabled, visual 1x impaired, learning disabled, hearing 
impaired) with three goals: 

1. Improve student academic performance and attitudes towards 
success in college through the use of computer technologx and 
academic skills training. 

2. Establish the Educational Center for Disabled Students 
utilizing appropriate computer equipment and software. 

3. Disseminate model project information concerning cc3mputer 
technologx and academic training to prospective students, 
parents, the business conmunitx and other post-secondarx 
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institutions. 

Thtst goaU sptcif itd tht direction that Ctnttr activititt would takt. 

Sinct tht Ctnttr was tstablishtd to ttst tht ftasibilitx of 
impl tmtnt ing a broad ttchiiologx bastd inttrvtntion program for ditabltd 
colltgt ttudtntt and to dtttrmint tht tf ftct ivtntss of such a program 
in tnhancing acadtmic ptrformanct, a comprthtnti vt tvaluatioii, 
including both format ivt and tumnativt atptcts was rtquirtd. As a 
dtmonstration projtct thtrt was a substantial disstmination compontnt 
that rtquirtd tht colltction and rtporting of a broad rangt of 
information to divtrst groups. 

Uhilt a substantial tvaluation nttd tx i sttd wi thin tht program, 
ftw rtsourcts wtrt allocittd to tvaluation activitits. Outsidt 
tvaluations had bttn tstabi ishtd wi th faculty at tht Univtrsity of 
Ntbraska to asstss attitudt changt and progrtss in writing with 
budgtt^ng for both consultant timt and graduatt assistants to work with 
thtst asptcts of tvaluation. No sptcif ic staffing and no funds wtrt 
allocattd within tht grant for othtr asptcts of tvaluation. Sinct tht 
Ctnttr was staff td with ont full -timt coordinator and two half-timt 
graduatt assistants who had administration, budgtting and program 
dtlivtry rtsponsibil i tits, thtrt was limittd ptrsonpowtr for conducting 
tvaluation activitits. Thus, any tvaluation would havt to bt dont 
within timt constraints impostd by tht small sizt of tht Ctnttr staff. 
Alsoi sinct tht program was tht first of its kind, thtrt was littit 
dirtction as to how tht ttchnoiogy could bt ustd and what wtrt 
rtasonabit outcomt txptctat ions. Tht tvaluation, thtrtfort, had to bt 
fitxibit tnough to providt dirtction, ytt still allow for pottntially 
substantial changts in program dirtction and outcomts. 
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Ut Do - Vhtx Do MM crtatfd to addrtss thtst tvaluation nteds and 
ovtrcoiftf tht constraints o^ liMittd tta^^ and financial rttourcts. 



Thf UE DO - THEY DO Modtl uses thrtt qutstions to organ izt 
fvaluation activititsi 

1. Did staff (Wf) do what wt said Mf wtrt going to do? 

2. Did participants (Thtx) do what thtx mrt suppostd to do? 

3. li fvfrxont did what thtx wtrt suppostd to dO| ho;* would wt 



Uhilf simplf and straightforward, thtst tht answtrs to thtst 
qutstions allow for a complttt tvaluation of both program activitits 
and outcomts. In addition thtse qutstions can bt answtrtd bx program 
staff from data gtntrated in tht courst of rtgular program activities. 



Tht first qutstion in the Model conctrns staff activitits, and 
asks whtthtr dtsigned activities were completed as specified. The 
following steps are used to answer the Ue Do question: 

1. Determine what activities must be done to meet the program 
goals. 

2. Determine how identified activities will be accomplished. 

3. Determine what documentation and reporting needs are related 
to staff activities* 

4. Determine how reporting needs will bt met. 

The first step requires the identification of program activities 
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that will be impltmenttd to rtalizt program goals. Goal stattmtnts for 
Mrvicf programs gtntrallv art sptcif ltd at hightr sxsttmt ItvtU or, 
as in thf cast of tht Educational Ctnttri are sptcifitd in funding 
grant proposals. Thtst goals art gtntrallx vagut and unsptcific as to 
itfhat actual acti vi tits i«i 1 1 bt dont in tht program. This ntctssitates 
specification of tht actual dax-to-dax activities that will nttd to bt 
dont if tht goal is to bt rtaliztd. 

Sttp 2 is to dfttrmint hw idtntifitd activitits will bt 
accompl ishfd. This sttp gtts at tht htart of tht Ut Do question by 
providing specifics on what will bt nttdtd to do tht stattd objtctives. 
At this sttp staff nttds to sptcifx txpts of inttrvtntions or 
instructional mtthods, curriculum, instrumtnts, and proctdurts that 
will impltment tht objtctivts. This sttp involvts optrat ional izing 
objtctivts into sptcific bthaviors and proctdurts. Esstntiallx, ttrms 
such as ttachi trtat, train, tvaluatt, asstss art dtf intd in ttrms of 
staff activitits. 

Sttp 3 of tht Ut Do qutstion is to dtttrmint what documtntat ion 
and rtporting nttds art rtlattd to staff activitits. Tht focus of this 
documtntation is not on outcomts; rathtr, it is on idtntifxing what 
information is nttdtd to know if staff activitits wtrt dont as 
sptcifitd. Gtntrallx, thtrt is ont rtporting nttd for tach staff 
activity idtntifitd in Sttp 2 of tht Ut Do qutstion. For txamplt, if a 
staff training) activity was to occur thtrt is a nttd to documtnt what 
took placti whtn tht training occurrtd, who atttndtd, and so forth. 
Thtrt may bt additional nttds to provide rtports such as monthly or 
quarttrly summaries of training or othtr activitits. Thtst types of 
rtports would allow staff to stt tht broad scopt of activitits and 

ERJC • 9 



Uf Do - Thf y Do 



anwtr qutstiont much as art training activitits balanctd or has 
training bttn ^ocustd on only tomt artas with othtrs bting ntgltcttd. 

Sttp 4 it to dtttriiinf how rtporting netds can b« mtt. This sttp 
involvfs idtntifxing instrumt nts and ntthods that can be ustd to 
colltct data. An important consideration at this sttp is tht timt and 
staff rtsourcfs that can bt allocattd to documtntation and rtporting. 
DocuMtntation of staff act iMi tits should not rtquirt txttnsive txtra 
work for staff. Thus, log shttts or activity rtports that providt 
rtcords of when and whert staff activities occurred art tht prtftrrtd 
rtporting r^t thod. 

Tht sfcond question in tht Modtl concerns client activitits, and 
asks whethtr clitnts ptrformtd tht activities assigned to thtm. A 
similar strits of sttps to thost in tht UE DO qutstion art ustd to 
assess clitnt activities as follows: 

1. Dfttrmint what activities clitnts art to tngagt in. 

2. Dfttrnint what documentation and reporting nttds art related 
to c1 ient activi ties. 

3. Determine how reporting needs will bt met. 

The first step requires identification of what activities clients 
will actually be doing in the cUss or program. As with staff 
activities there is a need to specify the types of day-to-day things 
clients will be doing. This is different from specifying client goals 
or objectives in a treatment plan or lesson plan. Most programs have 
client outccms specified; however, these behavioral or other 
objectives relate to the hoped-for result of staff and client 



9l0 



Wf Do - Thty Do 



ACtivitifs. Thftf objtctivts art likt tht goaU and objectives. 

While thex provide direction to activitiet, thex do not sax exactly 
what the client Mill do in the course of realizing the objective. 
Objectives must be translated into activities that will be done by the 
clients during the progrrn or class. The kex need in step one is to 
specif/ what clients will be doing with their time in the program. 

Step 2 is to identify documentation and reporting needs. As with 
staff activities, the focus here is on documenting that clients 
actually did the activities specified, rather than on documenting any 
outcomes of the activities. Key issues would be such things as: 

1. did clients attend scheduled sessions? 

2. did clients do practice activities? 

3. did clients complete homework? 

Again, there will generally be ore documentation need for each activity 
and additional summary reports. 

Step 3 IS to determine how reporting needs will be met. This 
requires identifying measures and instruments that can be u';ed to 
record client activities. As with staff activity reporting, most 
service or educational programs keep records on client activities such 
as case notes, attendance sheets, and records of assignment completion. 
Thusy Ue Do - They Do data collection need not appreciably increase the 
amount of recording required by implementation staff. The key aspect 
of the Ue Do - They Do Model is that case notes and other records 
should be directly tied to the specific activities clients will be 
doing in the course of the program. 
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Up to this point outcoiMt havt not b«fn tptcificallx addrtMtd in 
tht Ut Do - Thtx Do HodtK A Ktx point in tht Modtl it that outcomt 
qufstiont mutt follow activitx qutttiont. Dnix afttr %X^ii and client 
act iv it its havt bttn tptcif ltd and documtnttd is it appropriate to ask 
qutmtions concerning rttultt. Cltarlx thtrt mrt anticipated outcomts 
prior to tht tttabl i thmtnt oi a program that art givtn in grant 
prdposaU or in sxtttm levtl tpt cif icat iont. In tht Ut Do - Thtx Do 
iloui!; hOMVfr, outcomtt art rt-'txamintd and tptcifitd afttr staff and 
clitnt activitifs art dtttrmintd. Four sttps art ustd tht answtr the 
tf ftct i vtntss question as foil cms: 

!• Dtttrmint what outccmts can bt txptcttd to result from staff 
and c1 itnt activities. 

2. Determine how outcomes can be measured and what data exist 
that could be used to measure change resulting from program 
activities. 

3. Determine what methodologies can be utilized to assess the 
dat\. 

4. Determine what reports are needed to distribute outcome 
evaluation results. 

Step 1 in the effectiveness evaluation is the most critical and 
often the most inadequatelx ione in both program and sxttem wide 
evaluations. Step 1 involves examining staff and client activities 
that were identified and determin ing what changes in client behavior 
and what outcomes could be expected to occur if all activities were 
successfullx accomplished. While this seems logicallx apparent, often 
there is little direct relationship between what goes on in a program 
and the outcomes that are evaluated. For example, an often evaluated 
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outcoMf in dflinqutncx programs ior adoltftctntt is rtcidivism. Uhilt 
this i% obMiousIx a hoptd ^or goal i in manx casts thtrt is littlt 
dirtct rtlationship bftwttn program activitits and rtcidivifm. Uhilt 
fundtrsy sxsttm It^tl dtcision maktrsy and outsidt staktholdtrs max 
rtquirt that projtcts shOM impact on artas such as rtcidivism, 
impltmtntation ptrsonntl nttd to knom wht thtr thtir activitits wtrt 
succfssful at doing mhat thtx wtrt inttndtd to do. 

In thf Uf Do - Thtx Do Modtl tht proctss of tit ing outcomts to 
activitits btgins whtn staff activitits art sptcifitd in rtlation to 
program goal stattmtnts (which usual 1x contain broad, hoptd for 
outcomts). Afttr act iv; tits art sptcif itd| hoMtvtri possibit outcomts 
of thtst activitits art thtn idtntif itd. Tht tnd rtsult of this 
proctss is an txact spi^cif ication of outcomts that could bt txptcttd to 
rtsult from program activitits. Onct tht dirtct outcomts activitits 
art idtntif itd, indirtct outcomts can bt txamintd. Indirtct outcomts 
would bt things likt rtcidivism or ttst scort improvtmtnt that max 
rtsult ^v^nf» tnt program but do not ntctssarilx dirtctlx follow froni tht 
activitits dont. 

Sttp 2 involvts tht dtttrmi nation of how outcomts can bt mtasurtd 
and what data art availabit to asstss program outcomts. Sinct Wt Do - 
Thtx Do is dtsigntd to bt impltmtnttd at tht strvict providtr 1tvt1| 
mtasurts and data must bt obtainabit and impi tmtntabit bx strvict 
ptrsonntl within iimt and staffing constraints. As with documtntation 
of activititSi mtasurts and data art idtallx obtaintd from ongoing 
rtcords that art routintlx colltcttd. 

Uhtrt ntctssarx additional outcomt tvaluation data can bt 
colltcttd to suppltmtnt program activitx data. GtntrallXi this txpt of 
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inforiRation Mill bt nttdtd to addrtst indirtct outcoMs <f.g» 
rtcidiviM or improvtMnt in gradts). Tht impact of this data 
collfction on staff rttourctt can bt Ittsonitd by tstablithing 
rtasonablf tiM fraiift for colltcting this data and colltcting a 
ninimal aiaount of outtidt data. Sine* ongoing trtatiitnt dtla supplits 
information on tht immtdiatt rttultt of program activititti thtrt i% no 
rtaton to rtlx on indirtct goal data to documtnt activity 
tfffcti vtntM. Thtrtfort I data on indirtct outcomtt nttd onix bt 
colltcttd inf rtqutntlxi ovtr a long tnough timt framt for rtsults to bt 
appartnt. 

Sttp 3 i% to tttablith tht mtthodjiogi tt and rtstarch dtmigns 
notdtd to asMM tht data. An important asptct of Ut Do - Thtx Do is 
that a substantial amount of activity information is gtntrattd that 
allows for tvaluation bt/ond ttsting at tht start and tnd of a program. 
Rtgrtssion ttchniquts can possible bt ustd to rtlatt particular 
activitits to mtasurtd outccms or timt strits analysis max bt ustd to 
track changt across classts or counstling stMions. 

Sttp 4 is to idtntif/what rtports art nttdtd to distributt 
tvaluation data to staktholdtrs and hightr s/sttm-ltvcl ptrsonntl. 
I his sttp max varx considtrablx dtptnding on tht circumstancts of tht 
program. A classroom ttachtr max onix nttd to providt a sumnarx to tht 
school or district! whilt a grant fundtd program max havt to providt a 
formal rtport or publish rtsults. As with all Ut Do - Thty Do 
activitits rtport gtntration should bt ktr>t to a minimum and bt 
const sttnt with avail abit timt and staff rtsourcts. 
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Tht fvaluation o^ tht Educational Ctnttr iw Disabltd Studtnts 
will bt utfd to dtmonttrate how tht Ut Do - Thtx Do Modtl it 
impltifttntfd. In this txmplt i ont aiptct oi tht Educational Center's 
Qoalt will bf dfvf loptd into a Ut Do -Thtx Do evaluation. 

Goal 1 oi the Educational Center states that the Center will 
attempt toi 

Improve student academic performance and attitudes towards 
success in college through the use oi computer technology and 
academic skills training. 
While providing an overall direction to the progrMi this goal does not 
indicate how computer technology and skills training will be 
accomplished and thus, does not indicate specific staff activities. Ut 
Do procedures were applied to this goal to develop more specific 
objectives with the following objectives identified by staff and the 
advisory committees 

1. Evaluate student needs for adaptive hardware/software and 
skill training in academic areas. 

2. Provide training in adaptive hardware/software for areas 
identifieo in evaluation. 

3. Provide training in academic skill areas identified in 
evaluation. 

These objectives defined the types of staff activities that would be 
needed to meet goal one. The remainder of the application eximpU will 
focus on objective 2 

Step 2 in Ue Do involved operational izing these objectives into 
specific methods and training activities. Objective 2 stated that 
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staff would providf training in adapt ivt hardwart/tof twart for areas 
idtntifitd in tvaluation. To accomplish this objtctivt it was 
ntcfssarx to dtttrmint what hardware and software would be used in the 
Center, what types of training would be required for students to use 
the equipment (e.g. keyboarding, software operation, etc.), how 
training would be done (e.g. group or individual instruction), and what 
training mater iai should be used (e.g. in-house developed, packaged 
curriculum) . 

Once the training activities were identified. Step 3 reporting 
needs were considered. It was determined that critical reporting needs 
were: 

1. What equipment and software were staff training students to 
use. 

2. What training staff were doinc ard how often it occurred. 

3. What types of other student contact were staff doing and how 
often . 

4. What train ing mater iai s were being used. 

These reporting needs focused on how staff was implementing training 
and what typ^s of materials were being used to conduct training. 

Step 4 of Ue Do was implemented by identifying methods for 
accomplishing needed reports. Since reporting needs related to keeping 
track of client contacts, training sessions, and materials a staff log 
was developed allowing staff members to quickly indicate needed 
information on who was worked with, how long the contact lasted, what 
was done, and what was used. For formal group training sessions, a 
summary sheet was prepared to indicate attendance and what was done 
during training. 
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FollOMing tht sptci^ication oi activitits the Thtx Do 

qutstion was addrtsstd. Object ivts ^or each student were specified 
following an intake assessment. Typical objectives specified trainino 
in using particular software} using computer equipment, or training in 
educational skills. Once these objective were identified. Step 1 of 
The/ Do was addressed to specify student activities. For example, an 
objective stating that the student would learn to type to a level of 20 
words/minute was translated into activities such as: 

1. Client will attend an orientation session. 

2. Client will attend three typing instruction sessions. 

3. Client will spend 30 minutes per day doing self-instruction 
using the Typing Tutor program. 

In each case these activity objectives specify specific things the 
client will be expected to do. 

Step 2 involved identifying reporting needs. The primary 
reporting needs related to student activities were to know: 

1. Did students attend scheduled activities. 

2. Did students complete homework assignments. 

3. Uhat equipment and software were students using to do school 
work. 

4. Uhat kinds of school work were being done with the computers. 
Once these needs were identifies, Step 3 was addressed. To meet 
student reporting needs an activity log was again used. Students keep 
logs of their time using computers, other equipment and training 
materials both within the Center and outside the Center. These logs 
provide infc ion on client activities on a day-to-day basis. 
Information on attendance and homework completion was available on 
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%tkii 1ogt| so no additional rtporting Mat nttdtd in thtst artas. 

Onct both staff and studtnt activitits wtrt sptcifitdi 
tf f f ct i vtnt ss issufs wtrt addrtsstd. Tht first sttp in tht 
ffftct i vtntss tvaluation was to idtntify txptcttd outconts. In tht 
Educational Ctnttri staff activitits art oritnttd toward training 
computtr and softwart usagt, and inproving study and othtr acadtmic 
skills. For fxamplfi training in typing is providtd. A dirtct outcome 
of this training should bt incrtastd typing spttd. This is, thtrtforti 
tht outcomt litasurt for typing instruction. Similarlyi outconif«> wert 
sptcififd for tach staff and clitnt activity that would indicatt 
whfthtr tht activities wtrt successful. 

In the case of the Educational Center, indirect outcomes were such 
things as improved course grades, increased graduation rates for 
disabled students, increased credit hour loads, fewer dropouts, and 
more actaissions. These are all outccxnes that could be expected to 
follow from a program that provides traini.«g in academic skills and 
training in using technological equipment that allows disabled students 
to improve their writing and take less time to do school related tasks. 
It is important to note, however, that these outcomes do not directly 
result from program activities and that all training could be 
successful without affecting these measures. 

Step 2 was to determine measures and data. In the Educational 
Center extensive pre-post testing and ongoing assessment was used to 
generate data for evaluation of activities. For examplt, the 
computerized typing instruction program utilized saves information on 
errors and speed that can be printed. Information is, thus, available 
following each lesson or practice session on measures relevant to 
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iiiprovfd typing skill. Notttaking and study skills training naterials 
contain workshttts and ttsts that could bt gradtd and rtcordtd. In 
othtr casts ptrformanct ttsts wtrt ustd to asstss tht studtnts ability 
to optratt tht computtri load programs and run softwart. Gtntral cast 
notts Mtrt also ustd to rtcord tht Itvtl ptrformanct and sta^f 
imprtssions at tach stssion. 

Sttp 3 Mas to dtttrmint tht mtthodologi ts for asstssmtnt. For 
asstssmtnt oi dirtct outcomts a charting proctdurt is utiliztd. 
Ptrformanct at tach stssion is charttd and trtnds ovtr tint art 
analyztd by obstrving incrtasts or dtcrtasts in ptrformanct ovtr tint. 
Sinct tff tctivtntss tvaluation of activitits was for inttrnal use no 
statistical proctdurts havt bttn ustd to datt. For asstssmtnt of 
indirtct outcomts prt-post, bttwttn groups, and rtgrtssion dtsigns Mtre 
all tmploytd. Apptndix A providts and txampit of tht statistical 
dtsigns rtlattd to writing improvtantnt • 

Sttp 4 was to identify rtporting nttds. Sinct tht Educational 
Ctnttr was fundtd as a dtmonstration projtct a broad rangt of rtporting 
was rtquirtd. Reports to fundtrs wtrt nttdtd to summarizt activity 
outcomts and indirtct outcome results. Presentations and papers to 
professional groups were needed to provide information on 
interventions, equipment and training activities. Information on the 
program was to be distributed to schools and prospective students 
concerning available services and equipment. Finally, reports to the 
business community on potential worksite modifications for disabled 
workers were required. All of these reports could be compiled from 
available activity and outcome data generated through Ue Do - They Do 
evaluation. 
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This paptr has prtstnttd an ovtrview oi tht Ut Do - Thty Do Model 
ior strv icf program tvaluation. Tht Modtl is gtartd toward evaluation 
oi direct service delivery programs. The Model orients evaluation 
activities in relation to three questions: 

1. Did staff (Ue) do what we said we were going to do? 

2. Did participants (They) do what they were supposed to do? 

3. li everyone did what they were supposed to do, how would we 
know \i it did any good? 

These questions allow for evaluation of both program activities and 
program outcomes f^r both formative and sunmative purposes. Key 
aspects of the Model are a focus on evaluation of treatment integrity 
and a simple , straight forward methodology that can be implemented by 
service staff with minimal disruption of program activities. 
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APPENDIX A 



Program Goal 1.0 
Objective 1.6 



EDUCATIONAL CENTER FOR DISABLED STUDENTS 
SU^f1ATIVE EVALUATION 

Improve student academic performance and attitudes 

Evaluate progress In student writing resulting from use of adaptive hardware/software 



EVALUATION OBJECTIVE 



MEASUREMENT 

INSTRUMENTS BASELINE 



METHOD 



DATA COLLECTION 



SCHEDULE 



DESIGN 



DATA ANALYSIS 

GROUPS/MEASURES 



1. Increase writing 
abilities of stu- 
dents in center 
relative to dis- 
abled students 
not In center 



Writing samples 



Preliminary 
writing score 



Collect writing samples 
from students in center 
and group of disabled 
students not in center. 
Score pre and post 
samples blind using 
holistic scoring. 



Test at begin- 
ning of school 
year (Sept.} 
or at entry to 
prooram. 



Between groups la. 

pre - post 

comparison^ 

us1n^ last two lb. 

writing samples. 



Disabled students not 
in center (Control) 

Center students 
(Experimental) 



Determine degree 
of relationship 
between use of 
center equipment 
and improvement 
in writing 



Writing samples 
(Criterion) 
Use logs 
(Predictor) 



N/A 



Collect writing samples 
from students in center. 
Score pre and post 
samples blind using 
holistic scoring and 
compute difference 
score for each student. 
Compute total center 
use time and time on 
specific equipment 
from logs for each 
student. 



Test at begin- 
ning of school 
year (Sept.) 
or at entry to 
program for 
writing sample. 
End of school 
year for log 
siinmaiy. 



Regression of 
log data on 
writing dif- • 
ference scores 



la. Writing difference 
scores (Criterion) 

lb. Total use time for 
center (Predictor; 

Ic. Use time for spe- 
cific equipment 
(Predictor) 
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